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d • Abstract 



Several authors have examined connections among restricted permutations, contin- 
ued fractions, and Chebyshev polynomials of the second kind. In this paper we prove 
analogues of these results for involutions which avoid 3412. Our results include a re- 



I cursive procedure for computing the generating function for involutions which avoid 

' 3412 and any set of additional patterns. We use our results to give enumerations and 

. generating functions for involutions which avoid 3412 and various sets of additional 

[ patterns. In many cases we express these generating functions in terms of Chebyshev 

' polynomials of the second kind. 

, Keywords: Restricted permutation; restricted involution; pattern-avoiding permu- 

tation; pattern-avoiding involution; forbidden subsequence; continued fraction; Cheby- 
shev polynomial; Motzkin path 



ix^ ■ 1 Introduction and Notation 



Let Sn denote the set of permutations of {1, . . . , n}, written in one- line notation, and suppose 
TT G Sn- For alH, 1 < i < n, we write n{i) to denote the zth element of n. We say vr is an 
involution whenever 7r(7r(z)) = i for all z, 1 < i < n, and we write /„ to denote the set of 
involutions in Sn- Now suppose tv E Sn and a E Sk- We say a subsequence of n has type a 
whenever it has all of the same pairwise comparisons as a. For example, the subsequence 
2869 of the permutation 214538769 has type 1324. We say tt avoids a whenever vr contains no 
subsequence of type a. For example, the permutation 214538769 avoids 312 and 2413, but it 
has 2586 as a subsequence so it does not avoid 1243. If tt avoids a then cr is sometimes called 
a pattern or a forbidden subsequence and vr is sometimes called a restricted permutation or a 
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pattern- avoiding permutation. In this paper we will be interested in permutations (and often 
only involutions) which avoid several patterns, so for any set R of permutations we write 
Sn{R) (resp. /„(i?)) to denote the set of permutations (resp. involutions) in Sn which avoid 
every pattern in R and we write S{R) (resp. I{R)) to denote the set of all permutations 
(resp. involutions), including the empty permutation, which avoid every pattern in R. When 
-R = {tti, TTa, . . . , TT,,} we often write S'„(i?) = S'„(7ri, TTg, . . . , tt^), In{R) = /n(7ri, vrg, . . . , tt^), 
S{R) = S{7ri, 7r2, . . . , vr,.), and I{R) = /(tti, 712,..., vr^). 

As several authors have shown, generating functions for various subsets of 5(132) and 
S'(1243, 2143) have close connections with continued fractions and Chebyshev polynomials 
of the second kind, and can often be computed recursively. For example, Egge and Mansour 
|EMn3[ Thm. 5.4] have shown that 



7re5(1243,2143) k>l 1 Xi ^ 

2 lib ^JU <2^Ju '^Jb 

1 X1X2X3 - 2 ^ 

-L X\X'2X'^X^ ' ' ' 

where Tkiji) is the number of subsequences of type 12 . . . A; in tt. Egge and Mansour have 
also shown |KMn3[ Thm. 6.5] that 

E |5„(1243, 2143, 12 . . . A;)|x" = 1 + \ J. ^ (2) 

n=0 Uk-l 

where Un{x) is the nth Chebyshev polynomial of the second kind, which may be defined by 
sin((n + l)t) 

Unicost) = . Meanwhile, Krattenthaler has shown IKraOH Theorem 3] that 

sint 

Here the sum on the left is over all permutations in S'(132) which contain exactly r sub- 
sequences of type 12 ... A;, the quantity |7r| is the length of tt, and the sum on the right is 

h 

over all sequences li,l2, ■ ■ ■ Jb of nonnegative integers such that ^ ~ ^- Fiii^-Uyj 

i=l 

Mansour and Vainshtein have given |MVOH Thm. 2.1] the following recursive formula for 

oo 

/^(x) = J2 l'S'n(132, 7r)|x", which makes it possible to compute fTr{x) for any tt G S'(132). 

n=0 

r 

U{x) = l + xJ2 ( /-^ (^) - i^)) /-^ (x) (4) 

j=0 

Here tt-^"^, tt-^, and are the types of certain subsequences of n. For other results involving 
iS'(132) and continued fractions or Chebyshev polynomials, see |MV02j and the references 
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therein. For other results involving S'(1243, 2143) and continued fractions or Chebyshev 
polynomials, see |EM03j and |Rei03j . 

Involutions which avoid 3412 are known to have many properties which are analogous to 
properties of permutations which avoid 132. For instance, it is well known that |5'„(132)| = 
Cn for all n > 0, where C„ is the nth Catalan number, which may be defined by Co = 1 and 

n 

C^ = Y,Ci^lCn-^ {n>l). 

i=l 

(The Catalan number C„ may also be defined by C„ = ;^(^^)-) As a result, for all n > 0, 
the set S'„(132) is in bijection with the set of Dyck paths. These are the lattice paths from 
(0, 0) to (2n, 0) which contain only up (1, 1) and down (1, —1) steps and which do not pass 
below the line y = 0. Guibert |c/ui95l Rem. 4.28] has shown that |J„(3412)| = M„ for all 
n > 0, where Mn is the nth Motzkin number, which may be defined by Mq = 1 and 

n 

Mn = Mn-l + J2 M^-2Mn-i (n > 1). 

1=2 

As a result, for all n > 0, the set /„(3412) is in bijection with the set A4n of Motzkin paths. 
These are the lattice paths from (0,0) to (n, 0) which contain only up (1, 1), down (1, — 1), 
and level (1, 0) steps and which do not pass below the line y = 0. We write to denote the 
set of all Motzkin paths, including the empty path. (For a partial list of other combinatorial 
objects counted by the Motzkin numbers, see |Sta99t pp. 238-9].) 

Motivated by the parallels among 5(132), S'(1243, 2143), and J(3412), in this paper we 
prove analogues of (HJ, ©, (jH), and several similar results for /(3412). We begin with 
some results concerning J(3412) and continued fractions. We first define statistics Tk, k > 1, 
on A4 and /(3412). On /(3412), the statistic is simply the number of subsequences of type 
A; ... 21. On Ai, the statistic is a sum of binomial coefficients over the steps in the path. 
We then give a simple bijection Lp : A4 ^ /(3412) with the property that rfc((^(7r)) = rfc(7r) 
for all A; > 1 and all vr G Ai. Using 99 and a result of Flajolet, we prove the following 
analogue of (P). 



E n 



^e/(3412) k>l l-Xi ^ % , , 

1 _ ^ „2 X^X2X3X4 

± J/XX2X3 454 

-L X\X'2X^X^X^ ... 

2n /2n-2\_^/'2n-l\ 2n+l (' 2n N 

Here the nth numerator is H v '-1 g^^^ ^\^q ^^J;^ denominator is 1 — • 

■t=l i=l 

By specializing the XjS in (jSl) appropriately, we obtain continued fraction expansions of the 
generating functions for several other statistics on /„(3412), including inversions, left-to- right 
maxima, and right-to-left minima. 

We then turn our attention to analogues of ©, and (H)). We first use tp and some 
well-known results concerning lattice paths to find the generating function for the involutions 
in J(3412) which contain exactly r subsequences of type k . . . 21. This generating function 
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is an analogue of the generating function in 0, and we express it in terms of Chebyshev 

oo 

polynomials. We then find a recurrence relation for Ft{x) = Yl l-^n(3412, r)|x" which 

n=0 

enables us to compute Ft{x) for any set T of permutations. As a special case we have 

k 

F^{x) = 1 + xFp{x) + x'J2 (^OT7(^) - (6) 



i=l 



for any permutation ir. Here the various subscripts of F on the right are the types of certain 
subsequences of tt. This result is an analogue of (jH). 

Next we use (jH)) to compute for various tt. For example, we show that for all A; > 1, 

Uk-i (^) 

^-.-(^) - 1^ (^) 

and 

Uk-^ m + f/.-2 m 



F2k-1...2lW 



^{Ukm+u,.,c-^))- 

These results are analogues of ©. We also show that 

F[k,i]{x) = Fk+i k+i-i...2i{x), (9) 
where [/i, /2, • • • , Im] is the layered permutation given by 

m m 

[h,h, ■ ■ ■ jlm] = /l, /l — 1, . . . , 1, /2 + ^1, ^2 + ^1 — 1, • • • , ^1 + 1, • • • , hi /i — 1, • • • , — + 1- 

i=l i=l 

For m > 2 the generating function F^^^ does not reduce quite as nicely as it does 
when m = 2. Nevertheless, we conjecture that -F[Zi,. ..,/„] (2;) is symmetric in li, . . . ,lm for all 
m > 1 and all /i, . . . , Im > 1- This conjecture has been verified for m < 4 and k < 20, as 
well as for m = 5 and li < 11, using a Maple program. 

In the next two sections we give enumerations and generating functions for various sets 
of involutions in /(3412). For instance, we show that for all n > and all A; > 1, 

|/„(3412,4231,A; + 2 A; + 1 ...4321)| = Fk+i,n+i- 

Here ^ is the ^-generalized Fibonacci number, defined by „ = for n < 0, 1 = 1, 

k 

and Fk^n = Fk,n-i for all n > 1. We also give enumerations or generating functions for 

i=l 

1^(3412, cr) for all a of length five or less. 

We conclude the paper by mentioning several directions for future research. 



2 Two Families of Statistics and a Bijection 

In this section we give two infinite families of statistics, one on Ai and the other on /(3412). 
We then give a simple bijection between A4n and /,i(3412) which relates our two families of 
statistics. We begin by recalling the recursive structure of A4. 
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Definition 2.1 For any Motzkin paths tti and 1x2, we write 

TTi * 7r2 = U-KiDt:2- 
Proposition 2.2 (i) For all n>l, the map 

Mn-l ^ Mn 

TT 1-^ LtT 

is a bijection between M.n-i CLnd the set of Motzkin paths in M.^ which begin with a 
level step. 

(ii) For all n>2 and all j such that 2 < j < n, the map 

Mj-2 X Mn-j — > Mn 
(771,772) ^ 771*772 

is a bijection between M.j-2 x -Mn-j and the set of Motzkin paths in Al„ which begin 

with an up step and first return to the x-axis at x — j. 
Next we define our statistics on Motzkin paths. 
Definition 2.3 Suppose n is a Motzkin path. For any step s G vr, we write ht{s) to denote 
the height of s, which is the x-coordinate of the left-most point of s. For any k we write 



where the first sum on the right is over all up and level steps in 77 and the second sum on the 
right is over all down steps in 77. Here we use the convention that (^) = whenever k <Q 
or k > n. 

Example 2.4 7/77 = ULUUDLDDLUD then Ti{tt) = 11, T2{n) = 22, Ts{7r) = 27, Tiin) = 
19; T5(77) = 7, TqItt) — 1, and Tkiji) — for all other k. 

As we show next, the statistics are compatible with the recursive structure of M.. 

Proposition 2.5 (i) For all n>l, all k, and all 77 e M.n-i, 

rfc(L77)= (^^° +r,(77). (10) 
(ii) For all n>2, all j such that 2 < j < n, all k, all 77i e M.j-2, and all 772 € Mn-j) 

Tfc(77i * 772) = ( , ^ -,) + (, ) +Tfc_2(77i) +2rfc_i(77i) +rfc(77i) +rfe(772). (U) 



k-1 \k-l 



Proof, (i) This is immediate from the definition of r^. 
(ii) By definition of and 77i * 772 we have 
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Here the first sum on the right is over all up and level steps in tti and the second sum on the 
right is over all down steps in tti. Now (ii) follows from the fact that for all n > and all k, 

□ 

Wc now turn our attention to /(3412), beginning with its recursive structure. We start 
with some notation. 

Definition 2.6 For any permutation tt, we write \tt\ to denote the length of n. For any 
integer n and any sequence a of integers we write a'^'^ to denote the sequence obtained by 
adding n to every entry of a. 

Definition 2.7 For any 7ri,7r2 G /(3412) we write 

n^*Ti2 = Kil + 2 TT+i 1 41"^!+'. 

We have now defined * on both M. 'x M. and /(3412) x /(3412), but it will always be 
clear from the context which definition is intended. 

Proposition 2.8 (i) For all n>l, the map 

4_i(3412) 4(3412) 

TT I— > 1 TT"*"^ 

is a bijection between /n_i(3412) and the set of involutions in /„(3412) which begin 
with 1. 

(ii) For all n >0 and all j such that 2 < j < n, the map 

/,_2(3412) X 4_,(3412) 4(3412) 
(7ri,7r2) ^ 7ri*7r2 

is a bijection between /j_2(3412) x /„_j(3412) and the set of involutions in 4(3412) 
which begin with j. 

Proof, (i) This is immediate. 

(ii) It is sufficient to show that the image of the given map is contained in 4(3412) and 
that the map is one-to-one and onto. 

To see that the image of the given map is contained in 4(3412), suppose tti G /j_2(3412) 
and 7r2 G 4-i(3412). It is routine to verify that tti * 7r2 is an involution, so we show it avoids 
3412. Suppose by way of contradiction that abed is a subsequence of tti * 7r2 of type 3412. 
Since a > d, abed must be entirely contained in j,Tr^^, 1 or vr^''^^'^^. But avoids 3412, so 
abed is contained in j, n^^, 1. Since a < b, we must have a 7^ j, and since c < d, we must 
also have d 1, so abed is contained in tt^^. But this contradicts our assumption that tti 
avoids 3412, so TTi * 7r2 avoids 3412. 

To see that the given map is one-to-one and onto, suppose tt G 4(3412) begins with j. 
Since vr is an involution, we must also have 7r{j) = 1. Moreover, if there exists k > j such 
that k appears between j and 1 in tt then 7r{k) ^ k and the subsequence j, k, 1, 7r{k) has type 
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3412. Since tt avoids 3412, the entries between j and 1 in tt are 2, ... ,j — 1. It follows that 
there exist unique involutions tti G /j_2(3412) and G /„_j(3412) such that n = tti* 7T2, so 
the given map is one-to-one and onto, as desired. □ 

We now define our statistics on /(3412). 
Definition 2.9 For any positive integer k and any permutation tt, we write Tkiir) to denote 
the number of decreasing subsequences of length k in n. For notational convenience we set 
rfc(7r) = for all k<0. 

We have now defined on both Ai and /(3412), but it will always be clear from the 
context which definition is intended. As we show next, the statistics Tk are compatible with 
the recursive structure of J(3412). 

Proposition 2.10 (i) For all n > 1, all k, and all tt G Z„_i(3412) we have 

T.(lvr+^)=(^°^)+r.(vr). (12) 

(ii) For all n > all j such that 2 < j < n, all k, all tti G /j_2(3412), and all 712 G 
/„_j(3412) we have 

7-fe(7ri * TTs) = ( "^tI + I, \) +7-^-2(7^1) + 2rfc_i(7ri) +rfe(7ri) + rfc(7r2). (13) 



k~lj \k-l^ 

Proof, (i) This is immediate from the definition of r^. 

(ii) Observe that every decreasing subsequence of length at least two in tti * 712 is entirely 
contained in either |7ri| + 2 tt^^ 1 or tt^''^^'^^. With this in mind, (ii) is immediate from the 
definition of tti * 7r2. □ 

Next we introduce a bijection between and /„(3412) which is compatible with the 
statistics and the recursive structures of 7V1 and /(3412). 

Definition 2.11 For any n G A4n, we write (pin) to denote the permutation obtained as 
follows. Number the steps in vr from left to right with 1,2, ... ,n. For each up step at height 
k, find the first down step at height k + 1 to its right and switch the labels of the two steps. 
Then (p{7r) is the involution obtained by reading the resulting labels from left to right. 
Example 2.12 If tt = ULUDDLUD then (/?(7r) = 52431687. 
Remark The map 93 also appears in |Gui95| Rem. 4.28]. 



Proposition 2.13 For all n > 0, the map ip is a bijection between J„(3412) and M.n such 
that Tk{(p{7T)) = Tfc(7r) for all k and all tt & M.. 

Proof. Observe that if tti and TI2 are Motzkin paths then ip[LTii) = 1ip{tt)~^^ and ip{tti*tt2) = 
(p{Tii) * (p{tt2)- Arguing by induction on n, the result now follows from Propositions 12.21 12.51 
01 andlZTHl □ 

For all TT G A^„, let tt*"^ denote the Motzkin path in Ain obtained from tt by re- 
versing the order of the steps and switching the up and down steps. For example, if 
TT = ULUUDLUUDDLLDDLUD then tt^^ = UDLUULLUUDDLUDDLD. Geometri- 
cally, tt'^^ is the path obtained by refiecting tt over the line x = |. We conclude this section 
by showing that the map on /„(3412) which corresponds via Lp to the map tt i— tt^"^ is the 
well-known reverse complement map. 
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Proposition 2.14 For any n G Ai, let n^'^ be as in the paragraph above. For any tx G 
/„(3412), let tt'^^ denote the permutation obtained by reversing the order of the entries in tt 
and then replacing each entry i with n + 1 — i. Then V5(vr^^) = '^{t^Y'^ for all n E M.. 

Proof. For notational convenience we set a = {3 = Lp{n^'^), and n = 

Observe that for any i, 1 < i < n, we have a{i) = i if and only if the ith step of n is 
a level step. This happens if and only if the n + 1 — ith step of tt^'^ is a level step, which 
happens if and only ii f3{n + 1 — i) = n + 1 — i. It follows that a^"^ and f3 have the same fixed 
points. 

Similarly, for all i,j such that 1 < z < j < n we have a{i) = j and = z if and only 
if the zth step of vr is an up step and the jth step of tc is the corresponding down step. This 
happens if and only if the n + 1 — ith step of n'^'^ is a down step and the n + 1 — jth step of 
TT is the corresponding up step, which happens if and only if (3{n + 1 — i) = n + 1 — j and 
P{n + 1— j) = n + l — i. It follows that a^'^ and (3 have the same 2-cycles. 

Combining these observations, we find that the entries of a'^'^ are the same as the entries 
of /3, so a^'^ = /3, as desired. □ 



3 Continued Fractions 

In this section we combine our bijection (f with a result of Flajolet to express the generating 
function for J(3412) with respect to r^, A; > 1, as a continued fraction. By specializing the 
indeterminates in this result appropriately, we also obtain continued fraction expansions of 
the generating functions for several other statistics on /„(3412), including inversions, left-to- 
right maxima, and right-to-left minima. We begin by setting some notation. 
Definition 3.1 For any given expressions Ui {i > 0) and hi {i > 0) we write 

Oo Oi 02 os 

bo + hi + b2 + bs + "' 



to denote the infinite continued fraction 



ao 



ai 



bi + 



02 



Os 



03 + 



64 + ■ ■ ■ 

We use the corresponding notation for finite continued fractions. 

We now recall the relevant specialization of Flajolet's result. 
Theorem 3.2 (Flajolet \Fla8(j\ Theorem 1]) For all i >1, let Xt denote an indeterminate. 
Then we have 



1 /-v«2 ryt Q'lS ,y«4 ,yi 



n 4 ' 



11 I ^ xi — 1 — Xixlxs — 1 — XiX2x'^xfx5 — — 



ttGM k>l 



1 - n 

1=1 

(14) 
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Combining this with Proposition 12.131 we obtain the following result. 
Theorem 3.3 For all i > I, let Xi denote an indeterminate. Then we have 



n n 



X 



2n /2n-2\,(2n-l\ 

n { «-l ) + [. »-l ) 

_ _ i 

Tk(n) -L X1X2 XiX^X^Xi j=i 



^G/(3412)fc>l ^ ^1 ^ i XiX^X^X^X^ 1-11 4'" 



i=l 



(15) 

Using psp we can express the generating function for /„(3412, k . . . 21) as a finite con- 
tinued fraction. 

Corollary 3.4 For all k > 1 we have 



^|4(3412,2fc...21)|x" = - ••• (16) 

1 — X — 1 — X — 1 — X— — 1 — x 



n=0 

k—l terms 

and 



5^|4(3412,2fc + 1...21)K = - ■■• ^. (17) 

I — X — 1 — X — 1 — X— — 1 — X — 1 



n=0 



fc— 1 terms 



Proof. To prove set Xi = x, Xj = 1 for all z, 2 < z < 2k, and Xj = for all z > 2fc in 

The proof of (fT7j) is similar to the proof of (fTBj) . □ 

We can also use (fT3jl to express the generating function for /(3412) with respect to various 
statistics as a continued fraction. 

Corollary 3.5 For any permutation tt, let inv{7r) denote the number of inversions in tt. 
Then 

1 2 2 5 4n — S 

gj™(vr)^|7r| ^ ^ 1 ^ Q . . . 

1 — X — 1 — xo^ — 1 — xo^ — — 1 — X(/^" — 

7re/(3412) 1 -n \i 

Proof. In (jl5j) . set Xi = x, X2 = and Xj = 1 for all i > 3. □ 

Corollary 3.6 For any permutation it, let m{7c) denote the number of nonempty decreasing 
subsequences in tt. Then 



1 x'^q^ x^q^^ x^q'^''^" ^ 



m(7r) |7r| 



g^^3^-,^2) 1 — X — 1 — xg"* — 1 — xg^° — — 1 — xg^" — 

Proof. In ()15p. set Xi = xg and Xi = q for all z > 2. □ 

For our next application of ()15|) . recall that a left-to-right maximum in a permutation tt 
is an entry of vr which is greater than all of the entries to its left. Similarly, a right-to-left 
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minimum in n is an entry of tt which is less than all of the entries to its right. As we show 
next, if TT G /(3412) then the number of left-to-right maxima and the number of right-to-left 
minima in vr can be expressed in terms of the statistics r^. Combining this with (|T5|l . we 
obtain a continued fraction expansion of the generating function for J„(3412) with respect 
to left-to-right maxima or right-to-left minima. 

Proposition 3.7 For any permutation n, let lrmax{ir) denote the number of left-to-right 
maxima in tt and let rlmin{ir) denote the number of right-to-left minima in vr. Then for all 
n e /(3412), 



oo 

lrmax{T[) = rlmin{Ti) = > (— l)^^^rfe(7r). 



k=l 



Proof. Set r = ^^"^ Proposition 12 . 1 Ol to find that for all tti, G /(3412) we 



i=l 



have r(l7rj'""'^) = 1 + Tijii) and t{txi * 112) = 1 + t{tx2). It is routine to verify that the same 
relations hold when r is replaced with Irmax or rlmin. Using Proposition 12.81 the result 
now follows by induction on |7r|. □ 



Corollary 3.8 We have 



1 — X — 1 — X — 1 — X— — 1 — X — 

7re/(3412) 7re/(3412) 

Proof. In (IT3j) . set Xi = xq and Xi = q'^'^^' ^ for alH > 2 and use Proposition 13. 71 to simplify 
the result. □ 

For our final application of (fT^ . recall that z is a fixed point for a permutation tt whenever 
7r(z) = i. As we show next, if vr G /(3412) then the number of fixed points in tt can be 
expressed in terms of the statistics r^. Combining this with (jl5j) . we obtain a continued 
fraction expansion of the generating function for J„(3412) with respect to the number of 
fixed points. 

Proposition 3.9 For any permutation tt, let fix{TT) denote the number of fixed points in n. 
Then for all n G /(3412), 



Proof. This is similar to the proof of Proposition 13.71 □ 
Corollary 3.10 We have 

^ ^ ^ ^ . . . ^ 

^ 1 — xq — 1 — xq — 1 — xq— — 1 — xq — 

Proof. In fll5|) . set Xi = xq and Xi = g*-"^-*' ^ for alH > 2 and use Proposition 13. 91 to simplify 
the result. □ 
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We now turn our attention to the question of which statistics on J(3412) have generating 
functions which can be expressed as continued fractions hke the one in ()15|) . We begin by 
specifying which continued fractions we wish to consider. 

By a Motzkin continued fraction we mean a continued fraction of the form 



1 — mo — 1 — 7712 — 1 — — — 1 — — 

where rrii is a monic monomial in a given set of variables for all z > 0. Observe that if 
fi, f2, fs, ■ ■ ■ are (possibly infinite) linear combinations of the r^s with the property that 
each Tfc appears in only finitely many fi, then by specializing the XjS appropriately in (fTH|) 
we can express the generating function 

7rG/(3412) k>l 

as a Motzkin continued fraction. For example, when only /i is present, we have the following 
corollary of Theorem 13.31 

Corollary 3.11 Let Ai, A2, • • • denote nonnegative integers and let f denote the statistic 



on /(3412). Then 



k>l 



7re/(3412) 



I _ 1 3;^g/(^^) 
" 1 - - 1 - xg/(32i)-/(2i) 



^2g/(4321)-/(21) ^2^/{2n...21)-/(2n-2...21) 



1 _ 3;g/(54321)-/(4321) _ - 1 - a;g/(2n+1...21)-/(2n...21) _ 



Proof. In (|15|). set Xi = xq^^ and Xi = q^^ for alH > 2 and use the fact that 

f — 1 \ 

/(n...21)-/(n-1...21) = 5^r^ A, 

i=o ^ ^ ' 

for all n > 2 to simplify the result. □ 

With the same arguments used to prove |BCS021 Thm. 2] and |EM03[ Thm. 5.12], one 
can also prove the following result. 

Theorem 3.12 The set of Motzkin continued fractions is exactly the set of generating func- 
tions for countable families of statistics on /(3412) in which each statistic is a (possibly 
infinite) linear combination of the TkS and each Tk appears in only finitely many statistics. 
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4 Involutions Avoiding 3412 and Containing /c ... 21 

In this section we use our bijection Lp to find the generating function for the permutations 
in /(3412) which contain exactly r decreasing subsequences of length k. We express this 
generating function in terms of Chebyshev polynomials of the second kind, so we begin by 
recalling these polynomials. 

Definition 4.1 For all n > —1, we write Un{x) to denote the nth Chebyshev polynomial 

sin((n + l)t) 

of the second kind, which is defined by U-i(x) = and Un(cost) = for n > 0. 

smt 

These polynomials satisfy 

Unix) = 2xUn-l{x) - Un^2{x) {u > 1). (18) 

___ 2 — Jf^ 

We will find it useful to reformulate the recurrence in ()18j) . replacing x with — — to 
obtain 

1 — f 1 — x\ /I — 



Our main results in this section are the following, which give the promised generating 
function. 

Theorem 4.2 Fix r > 1, k > 1, and 6 > such that 

'2k + 2b + 2\ /2k + 2b\ f2k + 2b+l 
r < mm || ^, ^ l,!^, ,1 + 

Then 



2k -1 ■\2k-l \ 2k -I 



^ ^-lil 4+, + ;, )\ k ) ([7, (igt) )■'»+' ' 



j=0 



Here the sum on the left is over all involutions in /(3412) which contain exactly r subse- 
quences of type 2k . . .21. The sum on the right is over all sequences do, . . . ,db and lo, . . . ,lb 
of nonnegative integers such that 



r 



/ /2k + 2i-2\ /2k + 2i-l\\ A , /2k + 2i\ 
SHl 2.-1 )n 2.-1 ))^S'-(2.-l)- 



Throughout we adopt the convention that (q) = 1 and (_°'^) = for any integer a. 
Theorem 4.3 Fix r > 1, k > 1, and b > such that {^''^^^^) <r < {^'"^^k^^) ■ Th 



i=0 



' ^d, + d,^, +h-l\ /d, + k\ {Uk (^))'°"''°'' ^-l-do-lo + gi2d,+l,) 



(22) 
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Here the sum on the left is over all involutions in /(3412) which contain exactly r subse- 
quences of type 2k + 1 . . .21. The sum on the right is over all sequences do, . . . ,di, and 
Iq, . . . ,lb of nonnegative integers such that 



2A; + 2i + 1 
2k 



2k + 2i 
2k 



b 
i=0 



U 



2k + 2i 
2k 



(23) 



For notational convenience we set d^i = 1. Throughout we adopt the convention that (q) = 1 
and (_"^) = for any integer a. 

To prove these theorems, we first need to set some notation and recall some preliminary 
results. We begin with some matrices which will prove useful. 

Definition 4.4 For all k > we write Ak to denote the k + 1 by k + 1 tridiagonal matrix 
given by 



/ X 


X 








• 


• 








0\ 


X 


X 


X 





• 


• 














X 


X 


X 


• 


• 












A, 















■ 


X 


X 


X 

















■ 


■ 


X 


X 


X 


Vo 











■ 


■ 





X 


X J 



We write to denote the k + 1 by k + 1 tridiagonal matrix obtained by replacing the entry 
in the lower right corner of A^ with 0. We write Ck to denote the k + 1 by k + 1 tridiagonal 
matrix obtained by replacing the entry in the upper left corner of Ak with 0. 

The matrices Ak, Bk, and Ck are closely related to generating functions for various sets 
of Motzkin paths. To describe this relationship, we let A4{r,s,k) denote the set of lattice 
paths involving only up (1, 1), down (1, —1), and level (1,0) steps which begin at a point at 
height r, < r < k, end at a point at height s, < s < k, and do not cross the lines y = k 
and y = 0. Similarly, we let Af{r,s,k) denote the set of lattice paths in A4{r,s,k) which 
do not have any level steps at height k, and we let C(r, s, k) denote the set of lattice paths 
in A^(r, s, k) which do not have any level steps at height 0. Modifying the proof of |KranH 
Thm. A2] slightly, we find that 



and 



7reA^(T,s,A:) 
7reC'(r,s,fe) 



kl 



kl 



kl 



-l)'-+Met(J-y4fc;g,r) 
det(/ - Ak) 

4)'-+^ det(/-Bfc;g,r) 
det(/-5fc) 

-1)-+Met(/-Cfc;5,r) 
det(/-C,) 



(24) 



(25) 



(26) 
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Here |7r| is the number of steps in tt, I is the identity matrix of the appropriate size, and 
det(M; s, r) is the minor of the matrix M in which the sth row and rth column of M have 
been deleted. The determinants in (I24|l . (j25|l . and (j26|l can often be expressed in terms of 
Chebyshev polynomials of the second kind. For instance, arguing by induction on k we find 
that for all A; > 0, 

-'U,+r(^—^\=dei{I-A,) (27) 



2x 



and 



( Uk+i ( ^) + Uk {-^ ) ) = det(/ - Bk) = det(/ - C^). (28) 



We now prove Theorem 14.21 
Proof of Theorem \4.^ First observe that in view of Proposition 12. 131 the generating function 
on the left side of (j2(Jj) is the generating function for the set of Motzkin paths from (0, 0) to 
(n, 0) for which r2fc(7r) = r. To compute this generating function, observe that every Motzkin 
path TT with r2fc(7r) = r can be constructed by the following procedure in exactly one way. 

1. Choose do, . . . ,db and Iq, . . . Jb such that (f?T|) holds. Construct a sequence of down 
and level steps which contains exactly di down steps at height k + i and k level steps 
at height k + i for < i < b and which satisfies all of the following. 

(a) The step immediately preceeding a step at height j is either a down step at height 
j + 1 or less or a level step at height j or less. 

(b) All steps after the last down step at height j are at height j — 1 or less. 

(c) The sequence ends with a down step at height k. 

2. If the first step is at height k + j, insert j + 1 up steps before the first step. Similarly, 
after each step except the last, insert enough up steps to reach the height of the next 
level or down step. 

3. After each down step at height k except the last, insert an (possibly empty) upside- 
down Motzkin path of height at most k — 1. 

4. Before the first step insert a path from height to height k — 1 which does not exceed 
height k — 1. 

5. After the last step, insert a path from height A; — 1 to height which does not exceed 
height k — 1. 

Since the choice at each step is independent of the choices at the other steps, and since every 
sequence of choices results in a path of the type desired, the desired generating function is 
the product of the generating functions for each step. 

To compute the generating function for step 1, suppose we have fixed do,---,db and 

/o, . . . , h', then each of the resulting partial paths will have generating function x^^" . To 
count these paths, we construct them from the top down. That is, we first arrange the lb 
level steps at height k + b; there is one way to do this. We then place the db down steps 
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at height k + b so that one of these steps occurs after all of the diagonal steps. There are 
{'^''^k ^^) w^ys to do this. We then place the Ib-i level steps at height A; + 6 — 1 so that none 
of these steps immediately follows a diagonal step at height k + b. There are {'^''^''''_'^) ways 
to do this. Proceeding in this fashion, we find that the generating function for step 1 is equal 
to 

7=0 V di+i + li J \ k J 

where the sum on the left is over all sequences do, . . . ,dh and Iq, . . . Jbof nonnegative integers 
which satisfy pijl . In the path obtained after step 2 there is exactly one up step for every 
down step so the generating function for step 2 is equal to 

b 

. (30) 
Using and ^T7\ . we find that the generating function for step 3 is equal to 

do-l 

(31) 



^u, (^) 

and the generating functions for steps 4 and 5 are both equal to 



X 



^ (32) 



Taking the product of the quantities in (j^ . (jHUj) . (jHT|) and the square of the quantity in 
(IH^ . we obtain ((201), desired. □ 

The proof of Theorem 14.31 is similar to the proof of Theorem 14. 2| using , ()2(ij) , and 



Theorems 14.21 and 14.31 have several interesting special cases; we give four of them here. 
Corollary 4.5 For all k > 1, 



2 ' 



where the sum on the left is over all involutions in /(3412) which contain exactly one subse- 
quence of type 2k . . .21. 

Corollary 4.6 For all k > 1, 

E-'"- 



2 ' 



where the sum on the left is over all involutions in /(3412) which contain exactly one subse- 
quence of type 2k + 1 . . . 21. 
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Corollary 4.7 For all k >1, 



3 ' 



where the sum on the left is over all involutions in /(3412) which contain exactly two subse- 
quences of type 2k . . .21. 
Corollary 4.8 For all k>l, 



3 ' 



where the sum on the left is over all involutions in /(3412) which contain exactly two subse- 
quences of type 2k -\- 1 . . .21. 

We close this section by observing that for all r > 1, the number of involutions in 
/„(3412) which contain exactly r subsequences of type 321 can always be expressed in terms 
of Fibonacci numbers, since U2 (^^) + Ui (^^) = ^— fr^- For example, it follows from 
Corollaries 14 . 61 and 14 . 81 that the number of involutions in /„(3412) which contain exactly one 

subsequence of type 321 is — ^ ^—^ and the number of involutions in /„(3412) 

-9n^ -15n2 + 29n-6^ 

which contam exactly two subsequences of type 321 is in+i H -fn- 

25 50 



5 Restricted 341 2- Avoiding Involutions 

We now turn our attention to generating functions for involutions in J(3412) which avoid a 
set of additional patterns. For any set T of permutations we write 

OD 

Ft(x) =5^|4(3412,T)|x", 

n=0 

and we observe that -F0(z) = 0, Fi{x) = 1, and ^12(2:) = -^21(2;) = • In this section we 

1 — x 

give a recurrence relation which allows one to compute Ft{x) for any T. We begin with a 
method of decomposing permutations and a map on permutations. 

Definition 5.1 Fix n > 1. We call a permutation n ^ Sn complete whenever no initial 
segment of of length k, 1 < k < n, consists of the numbers 1,2, . . . , k. 

Observe that for every permutation tt there exists a unique sequence of complete permu- 

fc-i 

tations ai, . . . ,ak such that vr = 0:102 "3 • • ■ <^fc • ^^^^ situation we often 

abbreviate n = ai\a2 \ . . .\c(k- 

Definition 5.2 For any permutation n, we define W as follows. 
1. = an(iT = 0. 

^- //" Kl > 2 and there exists a permutation a such that tt = |7r|o"+^l then W = a. 
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3- If |7r| > 2, there exists a permutation a such that tt = \7r\a, and a does not end with 1 
then n = a. 

4- //" 1^1 ^ 2, there exists a permutation a such that n = cr^^l, and it does not begin with 
\tt\ then vf = cr. 

5- If |7r| > 2, TT does not begin with \tt\, and tt does not end with 1 then W = 71. 

Observe that if tt and a are permutations then |7r| + 2 n^^l avoids a if and only if n 
avoids a. 

In order to give our recurrence relation for Ft{x), we need to set some additional notation. 

Definition 5.3 Let T = {tti, . . . , Tr^} denote a set of permutations and fix complete per- 
mutations a*, 1 < i < m, 1 < j < ki, such that tTj = a*^|...|a^. . For all ii,...,i 
such that < ij < k^, let t'*^-^* = I all . . . \a} , . . . , aV^l . . . la™ | and let T"^'^* 

For any subset Y <Z {l,...,m}, set 



m 



^= Uwi---i"i-i> U wi-->i}- (33) 

jGY j<^Y,l<j<m 

We now describe how to find the generating function for permutations which contain 
patterns in one set while avoiding patterns in another set. 

Lemma 5.4 With reference to Deftnition \5.^ fix ii, . . . ,ijn such that 1 < ij < kj. Then the 
generating function for those permutations which contain every pattern in T-'^l_\ and 
avoid every pattern in T^'^^^ is 

J2 i-irPrAx). (34) 

yc{l,2,...,m} 

Proof. This follows by a routine inclusion-exclusion argument, since 5'(3412, TyJ C S'(3412, Ty^) 
whenever I2 ^ ^i- ^ 

We are now ready to give our recurrence relation for Ft{x). 
Theorem 5.5 With reference to Definition \5.'A 

ki,...,km I \ 

Ft{x) = 1 + xFp^T){x) + x^ V V (-I)I^IFt^(x) (x). (35) 

=1 \ yc{l,2,...,m} / 

Here Pi^iii) = tTj if a\ 7^ 1, (3{7ii) = a^l . . . if a\ = 1, and I3{T) is the set of permutations 
obtained by applying (3 to every element of T . 

Proof. The set /(3412, T) can be partitioned into three sets: the set Ai containing only the 
empty permutation, the set A2 of those involutions which begin with 1, and the set of 
those involutions which do not begin with 1. 
The generating function for is 1. 
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In view of Proposition lTHT i) . the generating function for A2 is xFj3{T){x), where /^(vrj) = ixi 
ii a\ 7^ 1, fiijii) = all . . - la],^ if a\ = 1, and f3(T) is the set of permutations obtained by 
applying (3 to every element of T. 

To obtain the generating function for A3, we first observe that in view of Proposition 
EHfii), all permutations in have the form ai * a2- Since each a* is complete, if ai * a2 
contains a subsequence of type a* then that subsequence is entirely contained in either 

\ai\ + 2 (Ti^l or cr^''^^''^^. As a result, the set of involutions which avoid 3412 and T and 
which do not begin with 1 can be partitioned into sets Bi^^,,,^i^, where Bi^^ is the set of such 
involutions in which ai contains t/^-^, • , but avoids Tl^f* and ctt avoids T^^ht ^ view 



of Lemma l5.4| the generating function for Bi-^ . is ^ (— l)l^lFTy(a:) 1 F^right (x). 

Yyc{l,2,...,m} ' '1' 

It follows that the generating function for is 



=1 \yc{l,2,...,m} 



T' 



Add the generating functions for Ai, A2 and A3 to obtain (jH^j) . □ 

The case of fl35p in which |r| = 1 will prove useful, so we single it out here. 

Corollary 5.6 Suppose it = ai\...\ak is a permutation, where ai,...,ak are complete. 
Then 

k 

F,{x) = 1 + xF^ix) +x'Yl (^^^^(^) - P^-^^^U^i^)) Fa.i...ia,{x). (36) 

1=1 

Here (3 = tt if ai 1 and /5 = 02! • ■ ■ l^fc ^/ «i = 1- 
Proof. Set T = {tt} in Theorem 15 .51 □ 

Observe that if 7f 7^ tt or tt is not complete then (jHUj) allows one to express F^(x) in terms 

of Fo-(x) for various a with \a\ < |7r|. U If = tv and tt is complete then one can solve (j36|) 

00 

to find that F^{x) = J2 |4(3412)|x". Since 4(3412,^) C /„(3412), we have the following 

n=0 

result, which can also be shown with a routine induction argument. 

Proposition 5.7 // 7f = vr and n is complete then /„(3412, tt) = J„(3412) for all n > 0. 

6 Generating Functions Involving Chebyshev Polyno- 
mials 

In this section we use to find F^(x) for various tt. In each case we express F^(x) in 
terms of Chebyshev polynomials of the second kind. We begin with Fk...2i{x). 

Theorem 6.1 For all k > 1, the following hold. 



^^-1 (^) 
^Uk (^) 



^2k.Mx) = ZU (37) 
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Proof. To prove (jHTj) . we argue by induction on k. It is routine to verify that the result 
holds for k = 1, so assume it holds for k — 1. Set tt = 2A; . . . 21 in (jHUj) and solve the resulting 
equation for F2k...2i{x) to obtain 

F2k...2lix) = 



1-X - x'^F2k-2...2l{x)' 

Now use induction to eliminate -^2^-2... 21 (2^) and simplify the result to obtain 



F2k...2lix) 



;i - x)Uk-^ (^) - xUk-2 (^) 



Finally, use p9|) to simplify the denominator and obtain (jSTj). 
The proof of (jHHj) is similar to the proof of (jHTj) . □ 

Remark Lines ()37p and ()38p can also be obtained using (f and the methods of Section HJ 

Arguing by induction as in the proof of (jH7|) and using Simion and Schmidt's result |SiS85| 
Prop. 6] that F3i2(x) = .^231(2^) = jE^^^ we obtain the following result. 
Theorem 6.2 For all k > 3, the following holds. 

Fk+l k..A23l{x) = Ffc...312(x) = Vl-.N (39) 

^Uk-1 (^) 

The fact that Fk+i k..A23i{x) = Ffc... 312(2;) implies that /„(3412, A; + 1 A;... 4231) and 
/„(3412, k . . . 312) have the same cardinality. But more is true: as we show next, /„(3412, k + 
Ik... 4231) and /„(3412, k... 312) are the same set. 

Theorem 6.3 Suppose a is a nonempty permutation which does not end with 1. Then for 
all n > 0, 

4(3412, |a| + 1 a) = 4(3412, |a| + 2 (t+^ 1). (40) 

Proof. It is clear that if tt avoids |cr| + 1 cr then tt avoids +2 a^^ 1, so 4(3412, |cr| + 1 a) C 
4(3412, |a| + 2 a+i 1). 

Now suppose TT contains |cr| + 1 cr; we show tt contains |cr| + 2 1. We argue by 
induction on |7r|. 

The result is vacuously true if |7r| < |a"|, and it follows for |7r| = |o"| + 1 since |cr| + 1 a does 
not end with 1, and so is not an involution. Now suppose the result holds for all involutions 
in J(3412) of length less than |7r|. If tt begins with 1 then the result follows by induction, 
in view of Proposition 12.81^ 1). If tt begins with j > 2 then by Proposition 12.8^ 11) there exist 
TTi G 4_2(3412) and 7^2 G 4-j(3412) such that tt = j Tif^ 1 tt^"'. Since + 1 ex begins with 
its greatest entry, it is entirely contained in either j ivf^ 1 or tt^"'. In the latter case the 
result follows by induction. In the former, |cr| + 1 cr is contained in j'tti, since cr does not end 
with 1. It follows that |cr| + 2 cr"*"^ 1 is contained in j Hi 1, as desired. □ 
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Corollary 6.4 For all n > and all k > A, the sets 4(3412, ... 312) and 4(3412, A; + 
1 . . . 4231) are equal. 

Proof. Set a = A; - 1 ... 312 in (gOI). □ 

We now turn our attention to Ffc...42i3(x), -Ffc 4132(2;), and -Ffc... 4123(3^)- 
Theorem 6.5 For all k > 3, the following holds. 

Fu 4213(X) = Ffc 4132(X) = ^^\^ ^^\^ (41) 

fc...4213l ; fc...4132l ; ^ > 

Proof. First observe that if we apply the reverse complement map to k . . . 4213 and take 
the inverse of the result we obtain k . . . 4132, so -Ffe... 4213(2^) = -^^...4132 (2;)- 

To show that Ffc... 4132(2;) is equal to the quantity on the right, first observe that the result 
holds for A; = 3 by |(4Mn21 Ex. 2.18]. Now argue by induction on k, using (1211) and (HH). ° 

Theorem 6.6 For all k > 3, the following holds. 

{1-X + x')Uk-3 (^) + (x - l)x/7,„4 (^) 



-^fc...4123(a;) 



:i-X + X^)xUk-2 (^) + (X - l)x2/7,_3 {^) 



Proof. This is similar to the second half of the proof of Theorem 16.51 □ 

Recall that 213 and 123 are examples of layered permutations, which are defined as 
follows. 

Definition 6.7 Fix n > 1 and let li,l2, ■ ■ ■ Jm denote a sequence such that li > 1 for 

m 

1 < i < m and ^ Zj = n. We write [li,l2, ■ ■ ■ , Im] to denote the permutation given by 

i=l 

[Zi, ^2, • • • , Zm] = /i, /i - 1, . . . , 1, ^2 + Zi, /2 + Zi - 1, . . . , /i + 1, . . . , n, n - 1, . . . , n - + 1- 

We call a permutation layered whenever it has the form [li, . . . , Im] for some sequence Zi , . . . , 

Observe that if m > 2 then [Zi, . . . , = [Zi, . . . , Z^,]. In view of (jHH), and the 
generating function F[/^^...^/^](a;) can be expressed in terms of Chebyshev polynomials of the 
second kind for any layered permutation [Zi, . . . , Z^]. For example, when m = 2 we have the 
following result. 

Theorem 6.8 For all k,l > 1 we have 

F[k,i]ix) = Fk+i k+i-i...2i{x). (42) 

To prove Theorem 16.81 we need the following well-known result concerning Chebyshev 
polynomials. 

Lemma 6.9 For all k,l > —1 and all w > we have 

Uk+wUi+w ~ UkUi = U^-iUk+i+w+i- (43) 
Here we abbreviate = (^i^)- 
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Proof. We argue by induction on k + w. 

Line (pSj) is immediate ioi k + w = —1 and k + w = so suppose k + w > and 
holds for k + w — 1 and k + w — 2. Divide ()19|1 by x and use the resuh to ehminate Uk+w 
and Uk on the left side of Now use induction and (fTIHl twice each to obtain □ 

We now prove Theorem 16.81 
Proof of Theorem \f).Sl We consider six cases: A; = 1 and / is even, k = 1 and / is odd, k and 
/ are both even, k is even and / is odd, > 1 is odd and / is even, and k > 1 and / are both 
odd. All six cases are similar, so we only give the details for the case in which k > 1 and I 
are both odd. 

Set TT = [2A; + 1, 2/ — 1] in and solve the resulting equation for F[2k+i,2i-i] i^) to obtain 

, , 1 - x'^Flou-i](x)Flo,-i](x) 



[2k-l]V-'^)^[2l-l]\ 

1 - X - x2F[2fc_i](x) - a;2F[2/_i](x) ' 



'[2k+l,2l-l][X) - - 



Now use to eliminate F[2k-i]{x) and F[2i-i]{x), clear denominators, and use (|T!?|) to 
simplify the denominator of the result, obtaining 

. , iUk + Uk-i){Ui + Ui-i) - {Uk^i + Uk^2){Ui-i + U1-2) 

[2.+l,2/-l] W ^^^^^^ ^ ^^^^^^ _^ _ ^^^^ ^ ^ . 

Here we abbreviate = {^^)- Next use fj43|l with w = 1 four times each in the 
numerator and denominator to obtain 

rp /' N _ ^'^+' 2f^fc+«-l + Uk+l-2 
i'[2k+l,2l-l\[.X) - —— — — — — -. 

X [Uk+i+1 + ^Uk+i + Uk+i-i) 

Rearranging (fTn|) we find that [/„ + [/„_i + f/n-2 = \Un-i for all n > 1. Use this to simplify 
our last expression for F]p,k+i,2i-i]{.x) , obtaining the right side of (jSZI) with k replaced by A; + /, 
as desired. □ 

When m > 3 the generating function Fj^^ , does not reduce quite as nicely as it 

does when m = 2. For example, using the same techniques as in the proof of Theorem 16.81 
one can prove that for all /ci, /c2, ^3 > 1, 

, N Uk-i^+k2+k3Uki+k2+k3-l + Uk-^+k2-lUki+ks~lUk2+k3-l 

i'[2ki,2k2,2k3][X) = — Yf U • 

XU ki+k2^ ki+ka'-J k2+k3 



Nevertheless, (P2|) and (ji^ suggest the following conjecture. 

Conjecture 6.10 For all m > 1 and all h, ■ ■ ■ ,lm ^ ^! the generating function F[;j_... 
is symmetric in li, . . . ,1^- 

This conjecture has been verified for m < 4 and k < 20, as well as for m = 5 and k < 11, 
using a Maple program. 



7 Involutions Which Avoid 3412 and Several Other 
Patterns 

Let T denote a set of permutations, each of which begins with its largest element and ends 
with 1 . In this section we use fl35|) to find the generating function for those involutions which 



21 



avoid 3412 and every element of T in terms of the generating function for those involutions 
which avoid 3412 and the permutations obtained by removing the first and last entry from 
each element of T. We then use this recurrence to enumerate involutions which avoid 3412 
and various sets of permutations. We begin by setting some notation. 

Definition 7.1 For any set T of permutations, we write kTl to denote the set of permuta- 
tions obtained by replacing each n & T with |7r| + 2 vr"'"^ 1. For all i > 2, we also write k^TV 
to denote the set obtained by applying this operation to T i times. 

Our main result, which we prove next, gives Fkxi{x) in terms of Ft{x). 

Proposition 7.2 For any set T of permutations, we have 

\ — X — x'^rTyx) 

Proof. Replace T with kTl in ()35p . use the fact that every permutation in kTl is complete 
and j3{kTl) = kTl to simplify the result, and solve for FkTi{x). □ 

Proposition 17.21 leads to an enumeration involving a generalization of the Fibonacci num- 
bers. 

Corollary 7.3 For all n > and all k > 1 we have 

14(3412,4231, fc + 2 A; + 1 . . . 4321)| = 
where Fk^n is the k-generalized Fibonacci number, defined by F^^n = for n < 0, F^ i = 1, 

k 

and Fk^n = Yl Fk,n-i for alln>l. 

i=l 

k—1 

Proof SetT= {k... 21, 12} in ^ and observe that Ft{x) = ^ x\ □ 

1=0 

If Ft{x) is a rational function then Ffcini(x) can be expressed in terms of Chebyshev 
polynomials of the second kind, as we show next. 

Corollary 7.4 Suppose Ft{x) = for polynomials /o and fi. Then for all i >1, 

fiix)Ui^i(^) - xfo(x)Ui^2(^) , . 

'^'^ ' xf,{x)U. {^) - x^fo{x)U.-i (^) ^ ' 

Proof. We argue by induction on i. 

When i = 1 the result is immediate from so we assume the result holds for i — 1. 
Replace T with /c*~^Tl*~^ in (jl^ . use induction to eliminate Ffci-iTiJ-i(x) on the right, and 
use (fT^ to simplify the result and obtain (jlUj) . □ 

Remark For any set T of permutations, let kT denote the set of permutations obtained by 
replacing each permutation n & T with |7r| + 1 vr. If no permutation in T ends with 1, then 
by Theorem 16.31 all of the results in this section hold when kTl is replaced with kT. 
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8 Enumerations 



In this section we give enumerations of /„(3412, cr) for various a; our results are listed in 
the tables below. We include references when they exist; if no reference is given then the 
enumeration follows from ()36|) . We place ai and ct2 on the same line whenever (T2 can 
be obtained from ai using the inverse and reverse complement maps, since in this case 
14(3412, (Ti) I = |/„(3412,a2)| for all n > 0. 

Several of our enumerations are expressed in terms of Motzkin, Fibonacci, or Pell num- 
bers. We write M„ to denote the nth Motzkin number, which may be defined by Mq = 1 

n 

and M„ = M„_i + ^ Mi_2Mn-i for all n > 1. We write F„ to denote the nth Fibonacci 

i=2 

number, which may be defined by Fq = 0, Fi = 1, and Fn = Fn-i + for all n > 2. 
We write P„ to denote the nth Pell number, which may be defined by Pq = 0, Pi = 1, and 
Pn = 2P„_i + P„_2 for all n > 2. 






14(3412, a)| 


Reference 


123 


2n2+7+(-l)" 
8 


Theorem 16.61 


132 


213 


Fn+l 


,(;Mn2| Kx. 2.18 


321 


Theorem 16.11 


231 


312 


2n-l 


|SiH85[ Prop. 6J 


1432 


3214 


Theorem 16.81 


2143 


Theorem 16.81 


4231 


Theorem 16.21 


4321 


Theorem 16.11 



23 



a 


14(3412, a)| 


Reference 


1234 


2n'*-4n^+28n2-2n+81-6n(-l)"+15{-l)" 




96 


1243 


2134 


5 4i+2 + -^Fn — -^Fn + 1 




1324 




1342 


1423 


2314 


3124 


p 1 — X — 

9-J- ~ l-2x-x'2+x'i 




2431 


4132 


3241 


4213 




21543 


32154 


Theorem 16.81 


43215 


15432 


Theorem 16.81 


53241 


52431 


Proposition 17^ 


42315 


15342 




54321 


Theorem 16.11 


52341 


^ J- -^J \^ } 


Proposition 17.21 


2341 


4123 


y-J ■ ~ l-3x+a;2+3a;3-3x4 


Theorem 16.61 


2413 


3142 




Proposition l^m 


3412 


|(;ui95l Hem. 4.28J 


3421 


4312 


Pn+Pn-l + 1 

2 


Theorem 16.21 


32541 


52143 




51432 


43251 


25431 


53214 




14352 


15324 


41325 


24315 




21534 


23154 


21453 


31254 




13542 


15243 


42135 


32415 




54231 


53421 


Proposition 17^ 
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a 




4(3412, a) 1 


Reference 


51324 


24351 




(l-x)(l-x-x^)^ 




32451 


51243 


52134 


23541 


9-f- - i_ 


4x+3x'^+4x''^-4:x'^-2x^+x^ 




21354 


3 ■ 2"-i - 


2F^^'\-\-nFr>^o-\-Fr,-i--\ A-nFn 




13254 


21435 


5 




24531 


51423 


53124 


34251 


9-J 


l—2x—x'^+x^ 




43521 


54213 


54132 


35421 




42351 


52314 


51342 


25341 


■ ~ l-3x+3x3 




34215 


15423 


43125 


14532 




12543 


32145 


(3n+7)2 
9 


+ 2 + 18( 1) 




14325 




15234 


13452 


23415 


41235 


9-f- = 


l~2x-x'^+3x^ -x'^-x^ 
l-3x+5xa-3x4-2x5+a:6 




12534 


12453 


23145 


31245 


9-f- = 


l-4x+3x2+4x3-4x*+a;S 




13425 


14235 


{l-x){l-2x-x'^+x'iY 




21345 


12354 


(5n2-3n-100)F„+i , (12n-38)F„ , 2n2+8n+23+{-l)" 




12435 


13245 


50 ' 25 ' 8 




54312 


45321 


jiF'n+l + — F2n-2) 


Theorem 16.21 


34521 


54123 


9-J- (i_2^)(i_3a;+2,.2+32:3_3^4) 


Theorem 16.61 


12345 


2n6_i2n5+86n"'-168n3+731n2-54n+1917+(-l)"(45n2-234n+387) 




2304 


51234 


23451 


f _ (1-^)5(1+^)2 
y-J- l-4x+3x2+6x=*-10a;4+6x0-4x'^ 





9 Directions for Future Research 

1. For any permutation tt 7^ A; ... 21, let 6*7^(0;) denote the generating function for the set 
of involutions which avoid 3412 and contain exactly one subsequence of type vr. Com- 
pute Gt^{x) or find a recursive formula for G^(a;). More generally, find the generating 
function for the set of involutions which avoid 3412 and contain exactly r subsequences 
of type vr. 

2. In view of ()37|) and ()39|) we have 

14(3412, 2A;. . .21)1 = |4(3412,A; + 2 A; + 1 . . . 4231)| 
for all ri > 0. Similarly, in view of (jHHj) and (^1]) we have 

14(3412, 2A;+ 1 .. .21)1 = |4(3412,A; + 2 A; + 1 . . . 4132)| 
for all n > 0. Give combinatorial proofs of these identities. 
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3. In view of (jl^ . we have 

14(3412, [A;,/]) I = |/„,(3412,A; + / fc + /-1...21)| 
for all k,l > 1 and all n > 0. Give a combinatorial proof of this identity. 

4. Prove Conjecture Ifi.KH which says that -Ff;^ /^](a;) is symmetric in /i, . . . , l^- 
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